Skip to content

Add kernel pinning and post-driver reboot to accel_drivers role#1204

Draft
skovili wants to merge 6 commits into
openstack-k8s-operators:mainfrom
skovili:accel-drivers-kernel-pin-reboot
Draft

Add kernel pinning and post-driver reboot to accel_drivers role#1204
skovili wants to merge 6 commits into
openstack-k8s-operators:mainfrom
skovili:accel-drivers-kernel-pin-reboot

Conversation

@skovili

@skovili skovili commented Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

  • Add kernel version pinning support to ensure driver/kernel compatibility for NVIDIA vGPU deployments
  • Add post-driver-install reboot with nvidia-vgpu-mgr verification for MIG instance detection
  • Both features are disabled by default and opt-in via configuration

Problem

MIG vGPU deployments on NVIDIA A30 GPUs require:

  1. A specific kernel version matching the NVIDIA AIE driver — without pinning, kernel updates break DKMS driver builds
  2. A reboot after driver installation so nvidia-vgpu-mgr can detect MIG instances and expose vGPU types via SR-IOV VFs

These were previously handled by custom OpenStackDataPlaneService CRs outside the role, which is not sustainable.

New Configuration

edpm_accel_drivers:
  kernel:
    pin: false           # enable kernel pinning
    version: ""          # e.g. "5.14.0-570.26.1.el9_6"
    remove_old: true     # remove non-target kernels
  reboot:
    post_driver: false   # reboot after driver install
    timeout: 1800
    wait_for_connection_timeout: 900
    verify_vgpu_mgr: true

Updated Role Flow

  1. setup_config.yml
  2. kernel_pin.yml ← NEW (when kernel.pin is true)
  3. driver_install.yml
  4. Cleanup temp dir
  5. boot_config.yml
  6. reboot.yml ← NEW (when reboot.post_driver is true)
  7. nvidia_mig_partitioning.yml

🤖 Generated with Claude Code

@skovili @claude
Add kernel pinning and post-driver reboot to accel_drivers role
63f5837 
MIG vGPU deployments require a specific kernel version matching the
NVIDIA driver and a reboot after driver installation for nvidia-vgpu-mgr
to detect MIG instances. These were previously handled by custom
OpenStackDataPlaneService CRs outside the role.

New features (disabled by default):
- kernel.pin: install a pinned kernel, remove old kernels, exclude
  kernel packages from future dnf updates
- reboot.post_driver: reboot after driver install and boot_config,
  wait for connection, verify nvidia-vgpu-mgr started

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci openshift-ci Bot requested review from abays and stuggi June 25, 2026 19:53
@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: skovili
Once this PR has been reviewed and has the lgtm label, please assign slagle for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Hi @skovili. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@skovili skovili marked this pull request as draft June 25, 2026 19:55
@skovili @claude
Address review findings: validation, idempotency, and tests
4561dbc 
- Add kernel version format validation with assert before any
  destructive operations
- Detect architecture dynamically via uname -m instead of hardcoding
  x86_64
- Fix dnf.conf exclude syntax (exclude= not excludepkgs=)
- Add idempotency guard to reboot: skip when running kernel already
  matches target
- Document kernel and reboot config in argument_specs.yml
- Add kernel_pin_reboot molecule test scenario

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@centosinfra-prod-github-app

centosinfra-prod-github-app Bot commented Jun 25, 2026

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/25172752d3ee4d3cac97e7bf098e9634

❌ openstack-k8s-operators-content-provider FAILURE in 7m 03s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 5m 07s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ noop SUCCESS in 0s
⚠️ edpm-ansible-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ edpm-ansible-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
❌ edpm-ansible-molecule-edpm_accel_drivers FAILURE in 21m 58s

@centosinfra-prod-github-app

centosinfra-prod-github-app Bot commented Jul 8, 2026

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/1910f93ddcb74d1a8bafd3f97f2ed154

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 51m 56s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 34m 20s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 44m 05s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 46s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 46m 39s
✔️ noop SUCCESS in 0s
✔️ edpm-ansible-tempest-multinode SUCCESS in 1h 46m 47s
✔️ edpm-ansible-baremetal-minor-update SUCCESS in 2h 38m 06s
❌ edpm-ansible-molecule-edpm_accel_drivers FAILURE in 19m 44s

skovili and others added 2 commits July 10, 2026 23:42
@skovili @claude
Fix kernel_pin_reboot molecule test for CI environments
fa14d27 
The molecule test was failing because:
- Hardcoded RHEL kernel version (5.14.0-570.26.1.el9_6) not available
  in CI repos — dnf install had nothing to do
- Mock grubby installed at /usr/bin but real grubby lives at /usr/sbin
- No /boot/vmlinuz-* files existed for grubby to operate on
- Reboot and nvidia-vgpu-mgr verification impossible in CI

Fix by:
- Dynamically detecting the installed kernel version in prepare and
  passing it to converge/verify via a shared file
- Installing mock grubby at /usr/sbin with symlink to /usr/bin
- Creating fake /boot/vmlinuz-* files for grubby mock
- Disabling reboot and vgpu-mgr verification (untestable in CI)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@skovili
Merge branch 'main' into accel-drivers-kernel-pin-reboot
cf14cdf
@skovili

skovili commented Jul 11, 2026

Copy link
Copy Markdown
Author

recheck

@centosinfra-prod-github-app

centosinfra-prod-github-app Bot commented Jul 11, 2026

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/db2f2089ac3a429ebd701d88cf9fece7

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 31m 10s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 33m 54s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 50m 20s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 24s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 54m 12s
✔️ noop SUCCESS in 0s
❌ edpm-ansible-tempest-multinode RETRY_LIMIT in 24m 36s
✔️ edpm-ansible-baremetal-minor-update SUCCESS in 2h 17m 20s
❌ edpm-ansible-molecule-edpm_accel_drivers FAILURE in 18m 17s

@skovili @claude
Fix molecule verify: mock grubby temp file permissions
9dae245 
The mock grubby script writes to a temp file via sudo (become: true),
which may create the file with restrictive permissions (0600) due to
sudo's umask override. The verify phase runs without become and cannot
read the file, causing grubby --default-kernel to return "none".

Fix by:
- Using /var/tmp instead of /tmp (not subject to tmpfiles.d cleanup)
- Explicitly chmod 644 the state file after writing
- Adding become: true to verify's grubby task to match converge

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@centosinfra-prod-github-app

centosinfra-prod-github-app Bot commented Jul 12, 2026

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/c7977fb014324086a5c7d7b4cd54e42b

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 00m 53s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 34m 14s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 44m 30s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 46s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 47m 46s
✔️ noop SUCCESS in 0s
✔️ edpm-ansible-tempest-multinode SUCCESS in 1h 46m 33s
❌ edpm-ansible-baremetal-minor-update FAILURE in 33m 15s
❌ edpm-ansible-molecule-edpm_accel_drivers FAILURE in 9m 51s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abays abays Awaiting requested review from abays
@stuggi stuggi Awaiting requested review from stuggi

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

do-not-merge/work-in-progress needs-ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@skovili